Flexible ethernet operations, administration, and maintenance systems and methods

ABSTRACT

A node configured to support a Flexible Ethernet (FlexE) client service in a network includes circuitry configured to interface a FlexE client; and circuitry configured to monitor and update one or more Operations, Administration, and Maintenance (OAM) fields associated with the FlexE client, wherein the one or more OAM fields include a monitoring field that covers 64b/66b codes for a path of the FlexE client. A method, implemented in a node, for supporting a Flexible Ethernet (FlexE) client service in a network includes interfacing a FlexE client; and updating and monitoring one or more Operations, Administration, and Maintenance (OAM) fields associated with the FlexE client, wherein the one or more OAM fields include a monitoring field that covers 64b/66b codes for a path of the FlexE client.

CROSS-REFERENCE TO RELATED APPLICATION(S)

The present patent/application is a continuation of U.S. patentapplication Ser. No. 14/824,667, filed Aug. 12, 2015, and entitled“FLEXIBLE ETHERNET OPERATIONS, ADMINISTRATION, AND MAINTENANCE SYSTEMSAND METHODS,” which claims priority to U.S. Provisional PatentApplication No. 62/186,502, filed Jun. 30, 2015, and entitled “FLEXIBLEETHERNET OPERATIONS, ADMINISTRATION, AND MAINTENANCE SYSTEMS ANDMETHODS,” the content of each is incorporated herein by reference.

FIELD OF THE DISCLOSURE

The present disclosure generally relates to networking systems andmethods. More particularly, the present disclosure relates to FlexibleEthernet (also referred to variously as Flex Ethernet or FlexE)Operations, Administration, and Maintenance (OAM) systems and methods.

BACKGROUND OF THE DISCLOSURE

Layer 1 protocols and technologies have evolved including SynchronousOptical Network (SONET)/Synchronous Digital Hierarchy (SDH) in the 1990sto Optical Transport Network (OTN) in the 2000s. SONET/SDH weresynchronous protocols optimized for circuit switching and transmission.OTN evolved from SONET/SDH to provide transparency and support forWavelength Division Multiplexing (WDM) as well as for optimizedtransmission of packet traffic. SONET, SDH, and OTN each have a richsuite of Operations, Administration, and Maintenance (OAM) functions andsupport for a wide range of services and applications. Conventionally,as OTN scales beyond 100G (B100G), there are emerging frameworks forLayer 1 functionality, namely Flexible OTN (FlexO or B100G) initiativesin the International Telecommunication Union (ITU) and Flex Ethernet inthe Optical Internetworking Forum (OIF).

Traditionally, Ethernet rates were defined in steps of 10×, i.e., 10Mb/s, 100 Mb/s, 1 Gb/s (GbE), etc. There is a wrinkle in this 10×progression where 40 Gb/s Ethernet (40 GbE) was defined. Today, thereare various Ethernet rates defined, including rates in-betweenestablished rates. IEEE 802.3 standards group is discussing 2.5 Gb/s, 5Gb/s, 25 Gb/s and other various odd rates. Specifically, different ratesare established for different applications, such as wirelessapplications, data center group applications, data centerinterconnections, etc. There is an expectation that different Ethernetrates will continue as new high-volume applications require optimizedsolutions. Specifically, router/switch equipment and opticaltransmission equipment are evolving at different rates. There is adesire to support simple transport of n×Ethernet streams across a fasterinterface. IEEE historically defines Ethernet rates (Media AccessControl (MAC) layer) with projects that also define the Physical(PHY)/Physical Medium Dependent (PMD) rates; the MAC rates and PMD ratesare tied and defined together. To address evolution in Ethernet anddissociate the MAC/client rate from the PHY/PMD, Flexible Ethernet hasbeen proposed. Note, as described herein, the terms Flexible Ethernet,Flex Ethernet, and FlexE can be used interchangeably.

In transport applications, FlexE can be used to match the flexibility ofoptical transmission equipment. Specifically, optical transmissionequipment (e.g., Dense Wave Division Multiplexing (DWDM)) is evolving tosupport variable modulation formats, Forward Error Correction (FEC)schemes, baud rates, etc. DWDM equipment can support a variable linerate with the same hardware, relying on configuration and provisioning.FlexE is based on Ethernet constructs, e.g., 64b/66b encoding,recognizing the primary client being transported is Ethernet. Note, thecurrent scope of FlexE is limited to interfacing applications (e.g.,bonding, subrating, and channelization). However, it may be advantageousto leverage Flexible Ethernet to augment or even replace OTN and/orFlexO in some transport and switching applications.

OAM functionality is critical for operating networks, fault isolation,service and path awareness, and the like. Again, OTN provides a rich setof OAM capabilities for layer 1 circuit switching and transportapplications. Ethernet also provides a rich set of OAM capabilities forlayer 2 packet networking and transport applications, such as throughthe efforts of the Metro Ethernet Forum (MEF). However, FlexE clients(in current IA) do not have defined OAM functionality and cannot betransported and switched as a service or private line service in aprovider's network.

BRIEF SUMMARY OF THE DISCLOSURE

In an embodiment, a node configured to support a Flexible Ethernet(FlexE) client service in a network includes circuitry configured tointerface a FlexE client; and circuitry configured to monitor and updateone or more Operations, Administration, and Maintenance (OAM) fieldsassociated with the FlexE client, wherein the one or more OAM fieldsinclude a monitoring field that covers 64b/66b codes for a path of theFlexE client. The monitoring field can provide an error check for theFlexE client instead of a Cyclic Redundancy Check (CRC) at a packetlevel. The monitoring field can utilize a bitstream Bit InterleavedParity (BIP) code. The one or more OAM fields can include a plurality ofa connectivity/trace, link fault, remote fault, maintenance states, testsignals, link neighbor discovery, and backwards fault and errorindication. The one or more OAM fields can have a different “O” codefrom standard Local Fault and Remote Fault information for an associatedFlexE group/PHY. The one or more OAM fields can use a multiframe schemewith the one or more OAM fields switched along with calendar slots overthe path. The one or more OAM fields can be located in the path in aReconciliation Sublayer. The one or more OAM fields can be located in aframe header in a FlexE shim. The node can be one of a RegeneratorSection (RS) Optical Transport Network (OTN) switch, a Multiplex Section(MS) OTN switch, and a FlexE switch.

In another embodiment, a method, implemented in a node, for supporting aFlexible Ethernet (FlexE) client service in a network includesinterfacing a FlexE client; and updating and monitoring one or moreOperations, Administration, and Maintenance (OAM) fields associated withthe FlexE client, wherein the one or more OAM fields include amonitoring field that covers 64b/66b codes for a path of the FlexEclient. The monitoring field can provide an error check for the FlexEclient instead of a Cyclic Redundancy Check (CRC) at a packet level. Themonitoring field can utilize a bitstream Bit Interleaved Parity (BIP)code. The one or more OAM fields can include a plurality of aconnectivity/trace, link fault, remote fault, maintenance states, testsignals, link neighbor discovery, and backwards fault and errorindication. The one or more OAM fields can have a different “O” codefrom standard Local Fault and Remote Fault information for an associatedFlexE group/PHY. The one or more OAM fields can use a multiframe schemewith the one or more OAM fields switched along with calendar slots overthe path. The one or more OAM fields can be located in the path in aReconciliation Sublayer. The one or more OAM fields can be located in aframe header in a FlexE shim. The node can be one of a RegeneratorSection (RS) Optical Transport Network (OTN) switch, a Multiplex Section(MS) OTN switch, and a FlexE switch.

In a further embodiment, a network supporting a Flexible Ethernet(FlexE) client includes a plurality of nodes interconnected to oneanother, wherein the FlexE client is configured between two nodes andtraverses one or more of the nodes; wherein each of the two nodesinclude circuitry configured to interface a FlexE client; and circuitryconfigured to monitor and update one or more Operations, Administration,and Maintenance (OAM) fields associated with the FlexE client, whereinthe one or more OAM fields include a monitoring field that covers64b/66b codes for a path of the FlexE client. The monitoring field canprovide an error check for the FlexE client instead of a CyclicRedundancy Check (CRC) at a packet level.

In an embodiment, a node configured to support a Flexible Ethernet(FlexE) client service in a network includes circuitry configured toreceive a FlexE client; and circuitry configured to at least one ofmonitor and update one or more Operations, Administration, andMaintenance (OAM) fields in FlexE overhead, wherein the OAM fields covera single client path for the FlexE client. The one or more OAM fieldscan include a client service error monitoring field which does not relyon packet Cyclic Redundancy Check (CRC). The one or more OAM fields caninclude a client service error monitoring field including a bitstreamBit Interleaved Parity (BIP) code. The one or more OAM fields caninclude any of a connectivity/trace, link fault, remote fault,maintenance states, test signals, link neighbor discovery, and backwardsfault and error indication. The one or more OAM fields can bedisassociated from standard Local Fault and Remote Fault information foran associated FlexE group/PHY. The one or more OAM fields can be locatedusing a multiframe scheme in the FlexE overhead. The one or more OAMfields can be implemented in a Reconciliation Sublayer. The node can beone of a Regenerator Section (RS) Optical Transport Network (OTN)switch, a Multiplex Section (MS) OTN switch, and a FlexE switch.

In another embodiment, a method, implemented in a node, for supporting aFlexible Ethernet (FlexE) client service in a network includes receivinga FlexE client; and at least one of updating and monitoring one or moreOperations, Administration, and Maintenance (OAM) fields in FlexEoverhead, wherein the OAM fields cover a single client path for theFlexE client. The one or more OAM fields can include a client serviceerror monitoring field which does not rely on packet Cyclic RedundancyCheck (CRC). The one or more OAM fields can include a client serviceerror monitoring field including a bitstream Bit Interleaved Parity(BIP) code. The one or more OAM fields can include any of aconnectivity/trace, link fault, remote fault, maintenance states, testsignals, link neighbor discovery, and backwards fault and errorindication. The one or more OAM fields can be disassociated fromstandard Local Fault and Remote Fault information for an associatedFlexE group/PHY. The one or more OAM fields can be located using amultiframe scheme in the FlexE overhead. The one or more OAM fields canbe implemented in a Reconciliation Sublayer. The node can be one of aRegenerator Section (RS) Optical Transport Network (OTN) switch, aMultiplex Section (MS) OTN switch, and a FlexE switch.

In a further embodiment, a network supporting a Flexible Ethernet(FlexE) client includes a plurality of nodes interconnected to oneanother, wherein the FlexE client is configured between two nodes andtraverses one or more of the nodes; wherein each of the two nodesinclude circuitry configured to receive the FlexE client, and circuitryconfigured to at least one of monitor and update one or more Operations,Administration, and Maintenance (OAM) fields in FlexE overhead, whereinthe OAM fields cover a single client path for the FlexE client. The oneof the two nodes can be a service terminating subscriber equipment andanother of the two nodes is a network element in a transport network,wherein the FlexE client includes a User-Network Interface (UNI). Eachof the two nodes can be a network element in a transport network, andeach of the two nodes is configured to operate as an OAM monitoringpoint for the FlexE client in the transport network. The one or more OAMfields can include any of a client service monitoring field, aconnectivity/trace, link fault, remote fault, maintenance states, testsignals, link neighbor discovery, and backwards fault and errorindication.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure is illustrated and described herein withreference to the various drawings, in which like reference numbers areused to denote like system components/method steps, as appropriate, andin which:

FIG. 1 is a diagram of a general structure of FlexE;

FIG. 2A is a block diagram of FlexE mux functions;

FIG. 2B is a block diagram of FlexE demux functions;

FIG. 3 is a diagram of a transport network aware or unaware of a routerto transport FlexE connection;

FIG. 4 is a network diagram of a transport network aware of FlexE;

FIG. 5 is a network diagram of a transport network of an example ofpartial-rate transport of Ethernet PHYs of a FlexE group;

FIG. 6 is a diagram of FlexE calendar distribution;

FIG. 7 is a diagram of FlexE overhead on each PHY of the FlexE group;

FIG. 8 is an example diagram of FlexE calendar slots where 25% of thecalendar slots are unavailable;

FIG. 9 is a diagram of encoding of ordered set block for FlexE overhead;

FIG. 10 is a diagram of FlexE overhead;

FIG. 11 is a diagram of data flow and interleaving for a FlexE clientmux in calendar slots;

FIG. 12 is a diagram of a standard Ethernet local fault ordered set;

FIG. 13 is a diagram of an Ethernet remote fault ordered set;

FIG. 14 is a diagram of data flow and de-interleaving for a FlexE clientdemux in calendar slots;

FIG. 15 is a diagram of a comparison between the layer stack and atomicfunctions of FlexO and FlexE;

FIG. 16 is a network diagram of a comparison of FlexO service OAM withthe FlexE service OAM systems and methods described herein;

FIG. 17 is a network diagram with various FlexE OAM monitoring points;

FIG. 18 is a diagram of a FlexE OAM implementation with FlexE overhead;

FIG. 19 is a block diagram of the FlexE overhead from FIG. 9 with a new“O” code for implementing FlexE OAM at the RS layer;

FIG. 20 is the logical diagram of FIG. 2 illustrating a location offirst and second options for locating FlexE OAM in the IEEE 802.3 stack;and

FIG. 21 is a flowchart of a process, implemented in a node, formonitoring a Flexible Ethernet (FlexE) client service in a network.

DETAILED DESCRIPTION OF THE DISCLOSURE

In various embodiments, FlexE OAM systems and methods are described toextend and use FlexE clients as a maintenance entity and service withdefined OAM functionality. The FlexE OAM systems and methods describedvarious different approaches to an OAM layer in FlexE client overhead.The FlexE OAM systems and methods also address transport performancemonitoring complexities introduced by FlexE including error checkingallowing monitoring and fault isolation of FlexE clients, without havingto perform packet-level error checking (i.e., MAC Frame Check Sequence(FCS)). There are markets (such as intra data-center) that could benefitfrom layer 1 techniques to bond, possibly switch large (elephant) flowsand bypass layer 2 switches. These markets have not adopted OTN for suchapplications. FlexE could be used in such markets to bypass layer 2switches, providing cost, power, latency, etc. benefits. Adding FlexEclient OAM would enable such application using FlexE client as a servicein a provider's network, and it would help with fault isolation anderror checking of variable number/size of flows.

Advantageously, the FlexE OAM systems and method augment/create theconcept/application of a FlexE client service with simplifiedperformance monitoring for transport equipment. The FlexE OAM systemsand methods add OAM fields to the FlexE client, including new LocalFault (LF)/Remote Fault (RF) client service fields independent of theFlexE group, which is tied to PHY LF/RF. The FlexE Client OAM providesfields in the FlexE frame overhead and/or in the path ReconciliationSublayer (RS); two approaches are considered.

Flexible Ethernet

FlexE provides a generic mechanism for supporting a variety of EthernetMedia Access Control (MAC) rates that may or may not correspond to anyexisting Ethernet PHY rate. This includes MAC rates that are bothgreater than (through bonding) and less than (through sub-rate andchannelization) the Ethernet PHY (Physical Layer) rates used to carryFlexE. This can be viewed as a generalization of the Multi-Link Gearboximplementation agreements, removing the restrictions on the number ofbonded PHYs (MLG2.0, for example, supports one or two 100GBASE-R PHYs)and the constraint that the client signals correspond to Ethernet rates(MLG2.0 supports only 10G and 40G clients). The Multi-Link Gearboximplementation agreements are described in IA # OIF-MLG-01.0 “Multi-linkGearbox Implementation Agreement” (May 2012) and IA # OIF-MLG-02.0“Multi-link Gearbox Implementation Agreement” (April 2013), the contentsof each are incorporated herein by reference.

Referring to FIG. 1, in an embodiment, a diagram illustrates a generalstructure of FlexE 10. Again, conventionally, the general capabilitiessupported by the FlexE include (i) bonding of Ethernet PHYs, e.g.,supporting a 200G MAC over two bonded 100GBASE-R PHYs, (ii) sub-rates ofEthernet PHYs, e.g., supporting a 50G MAC over a 100GBASE-R PHY, and(iii) channelization within a PHY or a group of bonded PHYs, e.g.,support a 150G and two 25G MACs over two bonded 100GBASE-R PHYs. Note,combinations are also contemplated, for example, a sub-rate of a groupof bonded PHYs, for example, a 250G MAC over three bonded 100GBASE-RPHYs.

The general structure of FlexE 10 includes a FlexE group 12, one or moreFlexE clients 14, and a FlexE shim 16. The FlexE group 12 refers to agroup including 1 to n bonded Ethernet PHYs. The FlexE group 12 caninclude one or more bonded 100GBASE-R PHYs as well as including otherrates, including new, higher rates once those standards are complete.The one or more FlexE clients 14 are each an Ethernet flow based on aMAC data rate that may or may not correspond to any Ethernet PHY rate.The FlexE client 14 MAC rates supported are 10, 40, and m×25 Gb/s. TheFlexE shim 16 is the layer that maps or demaps the FlexE clients 14carried over the FlexE group 12. Similar to the terminology of MLG, aFlexE mux refers to the transmit direction which maps the FlexE clients14 over the FlexE group 12. The FlexE demux refers to the receivedirection which demaps the FlexE clients 14 from the FlexE group 12.

Referring to FIGS. 2A and 2B, in an embodiment, block diagramsillustrate FlexE mux functions (FIG. 2A) and FlexE demux functions (FIG.2B). The FlexE shim 16 can be envisioned as being in the middle of thePhysical Coding Sublayer (PCS) in the 100GBASE-R stack as illustrated inFIG. 80-1 of IEEE Std 802.3-2015, the entirety of which is incorporatedherein by reference. The Media Independent Interface (MII),Reconciliation Sublayer (RS), and MAC above the FlexE shim 16 areseparate for each FlexE client 14 and operate at the FlexE client rate.The layers below the PCS (100GBASE-R Physical Medium Attachment (PMA),optional FEC, PMD) are used intact as specified for Ethernet. IEEE 802.3is a working group, and a collection of IEEE standards produced by theworking group defining the physical layer and data link layer's MAC ofwired Ethernet.

FIG. 2A illustrates the functions of the FlexE mux (the FlexE shim 16functions in the transmit direction). Where the 64b/66b encode and idleinsert/delete functions and whether these functions are part of theFlexE is application specific. What is presented for insertion into theslots of the FlexE master calendar is a stream of 64b/66b encoded blocksencoded per IEEE Std 802.3-2015 Table 82-4 which has been rate-matchedto other clients of the same FlexE shim 16. This stream of 66b blocksmight be created directly at the required rate using back-pressure froma Network Processing Unit (NPU). It might come from a single-laneEthernet PHY such as 10G or 25G, where the process of rate-matchinginvolves both idle insertion/deletion, plus converting the rate-alignedstream from the 4-byte alignment of IEEE Std 802.3-2015 clause 49 to the8-byte alignment of IEEE Std 802.3-2015 clause 82. Note that the IEEE802.3 diagrammatic convention of showing idle insertion/deletion asthough this were an operation that operates on a stream of 64b/66bblocks, even though strictly speaking this may require 64b/66b decodingand recoding, particularly in the case of converting between 4-bytealignment and 8-byte alignment. The stream of blocks may come from amulti-lane Ethernet PHY, where the lanes need to be deskewed andre-interleaved with alignment markers removed prior to performing idleinsertion/deletion to rate match with other clients of the same FlexEshim 16. Or the stream may have come from another FlexE shim 16, forexample, connected across an OTN network, where all that is required isto perform idle insertion/deletion to rate match with other clients ofthe same FlexE shim 16.

The 66b blocks from each FlexE client 14 are distributed sequentiallyinto the master calendar in the order described herein. The 66b blocksfrom the master calendar are distributed to each PHY of the FlexE group12 according to the ordering described herein. The FlexE overhead isinserted into the sub-calendar of each PHY. The stream of 66b blocks ofeach PHY is distributed to the PCS lanes of that PHY with the insertionof alignment markers, and this is presented at the PMA service interfacein the 100GBASE-R stack. Lower layers and interfaces of the 100GBASE-REthernet PHY (e.g., CAUI, FEC, PMA, PMD) are used as specified in IEEEStd 802.3-2015. Error Control blocks are generated for insertion intocalendar slots that are unused or unavailable. LF blocks are generatedfor insertion into the calendar slots of any failed FlexE client.

FIG. 2B illustrates the functions of the FlexE demux (the FlexE shim 16in the receive direction). The layers of each 100GBASE R PHYs below thePCS are used as specified in IEEE Std 802.3-2012, the entirety of whichis incorporated herein by reference. The PCS lanes are recovered,deskewed, reinterleaved, and the alignment markers are removed. Theaggregate stream is descrambled. The calendar slots of the each PHY arelogically interleaved in the order specified herein. The FlexE overheadis recovered from each PHY. In the case that any PHY of the FlexE group12 has failed (PCS_Status=FALSE) or overhead lock or calendar lock hasnot been achieved on the overhead of any of the PHYs, LF is generated tobe demapped from the master calendar for each FlexE PHY. The 66b blocksare extracted from the master calendar positions assigned to each FlexEclient in the order described herein.

Where the Idle Insertion/Deletion, 66B Decoding functions are performedand whether they are inside or outside the FlexE is applicationspecific. The 66b blocks could be delivered directly to an NPU. Ifdelivered to a single-lane PHY, idle insertion/deletion may be used toincrease the rate to the PHY rate, realigning to 4-byte boundaries inthe process (for 10G or 25G) and recoding 64b/66b according to clause49. For a multi-lane PHY, idle insertion/deletion is used to increasethe rate to the PHY rate less the space needed for alignment markers,the blocks are distributed to PCS lanes with AM insertion. For a FlexEclient mapped over OTN, idle insertion/deletion may be used to adjustthe rate as required for the OTN mapping.

FlexE Applications

FlexE can support a variety of applications. A non-exhaustive listincludes:

-   -   Router to Transport Connection.    -   Intra-Data Center “Fat Pipe” application: bonded PHYs for flows        exceeding the PHY rate, or carrying traffic that doesn't        distribute efficiently with LAG.    -   Generalized MLG for port expansion applications, e.g., an n×100G        PHY as an umbilicus to a satellite shelf of lower rate ports.

Referring to FIG. 3, in an embodiment, a network diagram illustrates atransport network 20 unaware of a router to transport FlexE connection.In FIG. 3, the FlexE shim 16, e.g., in a router or switch, maps theFlexE client(s) 14 over a group of bonded Ethernet PHYs 22. Each of theEthernet PHYs 22 is carried independently over the transport network 20using a PCS codeword transparent mapping (e.g., the mapping of100GBASE-R into Optical channel Path Unit 4 (OPU4) as described in ITU-TRecommendation G.709 clause 17.7.5), the entirety of which isincorporated herein by reference. The Ethernet PHYs 22 are intended tobe carried over the same fiber route. All of the PHYs 22 of the FlexEgroup 12 need to be interconnected between the same two FlexE shims 16.In this case, the FlexE shim 16 will need to tolerate and accommodateconsiderably more skew than if the FlexE shims 16 were only separated byan Ethernet link distance of 40 km or less, as the transport network 20could carry the signal over thousands of kilometers. For the caseillustrated in FIG. 3, it is the PHYs 22 of the FlexE group 12 which arecarried over the transport network 20. Note, in FIG. 3, the transportnetwork 20 is unaware of FlexE 10.

Referring to FIG. 4, in an embodiment, a network diagram illustrates atransport network 30 aware of FlexE. In FIG. 4, the distance between anypair of FlexE shims 16 is limited to the Ethernet link distance (about40 km maximum), so the amount of skew that needs to be tolerated andcompensated is considerably less. The other important distinction hereis that it is the FlexE clients 14 rather than the PHYs 22 of the FlexEgroup 12 which are carried over the transport network 30. The FlexEclient 14 could be constructed to be the complete size of the payloadthat can be carried over a single wavelength (e.g., construct 200G tofill a Dual Polarization (DP) 16-Quadrature Amplitude Modulation (16QAM)wavelength with the bonding of two 100GBASE-R PHYs), or could be asmaller client which is multiplexed and switched at a sub-wavelengthlevel, using the mapping described in ITU-T Recommendation G.709 clause17.9 to carry the FlexE client signal over an Optical channel Data Unitflex (ODUflex) Constant Bit Rate (CBR).

Referring to FIG. 5, in an embodiment, a network diagram illustrates atransport network 40 of an example of partial-rate transport of EthernetPHYs 22 of a FlexE group 12. Here, the Ethernet PHY rate may be greaterthan the wavelength rate, and there is a reason (for example,wavelengths terminated on different line cards) that it is not desirableto terminate the FlexE shim 16 in transport equipment 42. The exampleillustrated here is a 150G FlexE client carried over two 75Gwavelengths, but one can imagine a possible future 200 GbE rate whereone would want to carry a 300G FlexE client over two bonded 200 GbE andbe able to carry the two partially-filled Ethernet PHYs over 150Gwavelengths.

FlexE Group

The FlexE group 12 includes from 1 to n 100GBASE-R Ethernet PHYs 22.Each PHY can be identified by a number in the range [1-254]. The valuesof 0 and 255 are reserved. A PHY number may correspond to the physicalport ordering on equipment, but the FlexE shim 16 at each end of thegroup must identify each PHY in the group 12 using the same PHY number.PHY numbers do not need to be contiguous: even if a FlexE group 12 iscomposed of initially contiguous PHY numbers, this may not be preservedas PHYs are added to or removed from the group 12.

Each of these PHYs 22 uses the bulk of the PCS functions described inIEEE Std 802.3-2015 clause 82 including PCS lane distribution, lanemarker insertion, alignment, and deskew. All the PHYs 22 of the FlexEgroup 12 must use the same physical layer clock. Each PHY 22 of theFlexE group 12 is able to deliver a logically serial stream of 64b/66bencoded blocks from the FlexE mux to the FlexE demux at a data rate of:

${103.125\mspace{14mu} {Gb}\text{/}s \times \frac{16383}{16384}} \pm {100\mspace{14mu} {ppm}}$

FlexE supports a value of n (number of PHYs 22 in the FlexE group 12) upto 256, although practical implementations are likely limited to therange of 4-8 PHYs 22. The fraction applied to the base rate reflects thefact that 1/16K of the space of the interface is occupied by PCS lanealignment markers which are not space available to carry the FlexEpayload. The FlexE payload carried over each PHY 22 of the FlexE group12 has the format of a logically serial stream of (mostly) legal 64b/66bblocks with the format described in IEEE Std 802.3-2015 FIG. 82-4,although the blocks do not appear in a sequence that makes sense tointerpret as an Ethernet interface. The actual PHYs 22 of the FlexEgroup 12 may transcode these blocks to 256b/257b format according toIEEE Std 802.3-2015 clause 91.5.2.5 according to the PHY type, but theyare trans-decoded back to 64b/66b blocks prior to delivery to the FlexEdemux.

FlexE Client

Each FlexE client 14 is a logically serial stream of 64b/66b blocksrepresenting an Ethernet MAC layer. The MAC layer of a FlexE client 14may operate at a rate of 10, 40, or n×25 Gb/s. The 64b/66b encoding isaccording to IEEE Std 802.3-2015 FIG. 82-4. For certain cases where theFlexE client 14 originates from an Ethernet PHY, this may requireconversion from the 4-byte alignment of start-of-packet for a PHY whichuses the clause-49 PCS to the 8-byte alignment of FIG. 82-4 as part ofthe idle insertion/deletion process.

In the case where the FlexE client 14 comes from an Ethernet PHY whichuses PCS lane alignment markers (e.g., 40GBASE-R), the lanes must bedeskewed, re-interleaved and serialized, removing the alignment markersto produce the 64b/66b stream which is treated as a FlexE client 14. AllFlexE clients 14 transmitted over the same FlexE group 12 must bealigned to a common clock. This is accomplished using idleinsertion/deletion as described in IEEE Std 802.3-2015 clause 82.2.3.6.In addition, the bit-rate of each FlexE client 14 is reduced slightlyfrom nominal as part of this process to allow room for insertion ofFlexE overhead and the PCS lane alignment markers of the FlexE group 12.So the 64b/66b encoded format of a FlexE client 14 operates at a datarate of:

${{FlexE}\mspace{14mu} {Client}\mspace{14mu} {MAC}\mspace{14mu} {rate} \times \frac{66}{64} \times \frac{16383}{16384} \times \frac{20460}{20461}} \pm {100\mspace{14mu} {ppm}}$

This is a rate reduction slightly under 0.011%, which is well withinwhat can be accomplished with idle insertion/deletion without packetloss. Note that this does not actually correspond to any clock thatneeds to be generated in an implementation, as the idleinsertion-deletion process will simply operate by filling the allocatedblock positions in the FlexE group 12 from a FlexE client 14 First-InFirst-Out (FIFO) with its own deficit idle counter and inserting ordeleting idles in the process of filling the block positions in theFlexE group 12 according to a FlexE calendar.

FlexE Calendar

The FlexE operates using a calendar which assigns 66b block positions oneach PHY 22 of the FlexE group 12 to each of the FlexE clients 14. Thecalendar has a granularity of 5G and has a length of 20 slots per 100Gof FlexE group 12 capacity. Two calendars are supported: an “A” and a“B” calendar. At any given time, one of the calendars is used formapping the FlexE clients 14 into the FlexE group 12 and demapping theFlexE clients 14 from the FlexE group 12. The two calendars are providedto facilitate reconfiguration.

For a FlexE group 12 including n bonded 100GBASE-R PHYs, the logicallength of the master calendar is 20n. The blocks as allocated per themaster calendar are distributed to n sub-calendars of length 20 on eachof the PHYs of the FlexE group as shown in FIG. 6. FIG. 6 is a diagramillustrating FlexE calendar distribution. The order of distribution oftwenty blocks at a time is selected over simple “round robin”distribution of 66b blocks to facilitate addition of PHYs 22 to a FlexEgroup 12 without the need to change the calendar slots allocated toexisting FlexE clients 14. Calendar slots are identified by their PHYnumber and the slot [0-19] (within that PHY). The calendar slotsassigned to FlexE clients 14 do not change when PHYs are added orremoved from the group 12. PHYs with calendar slots assigned to FlexEclients 14 cannot be removed from the group (or all FlexE clients 14need to be moved to other PHYs or removed BEFORE a PHY is removed fromthe group). The “logical” sequence number of a calendar slot is 20×thePHY number plus the calendar slot number within the PHY. The sequence isascending order. Note that the sequence numbering is not necessarilyconsecutive when the assigned PHY numbers are not contiguous. Thislogical order only matters when calendar slots on different PHYs areassigned to the same FlexE client 14.

FlexE Overhead and Alignment

The alignment of the data from the PHYs 22 of the FlexE group 12 isaccomplished by the insertion of FlexE overhead into a stream of 66bblocks carried over the FlexE group 12. The FlexE overhead is encoded asa 66b block which can be recognized independently of the FlexE clientdata. An illustration of the FlexE overhead on each PHY of the FlexEgroup is shown in FIG. 7.

On a 100GBASE-R PHY, a FlexE overhead block will occur approximatelyonce per 13.1 μs. The actual format of the FlexE overhead blocks is suchthat they occur in a repeating sequence of four blocks, so the sequencehas a period of approximately 52.4 μs. This sequence is used to alignall of the PHYs 22 of the FlexE group 12 at the FlexE demux toreconstruct the sequence in the order of the master calendar so that theFlexE clients 14 can be recovered.

The scenario illustrated in FIG. 5 is supported by marking a certainnumber of the calendar slots as unavailable. This is different from“unused”, in that it is known, due to transport network constraints,that not all of the calendar slots generated from the FlexE mux willreach the FlexE demux and, therefore, no FlexE client 14 should beassigned to those slots. The intention is that when a PHY 22 of theFlexE group 12 is carried across the transport network, the mapping isable to compress the signal to less than the PHY rate by dropping theunavailable calendar slots. A case where 25% of the calendar slots areunavailable is illustrated in FIG. 8.

The anchor position FlexE overhead is encoded as an ordered set (controlblock type 0x4B). A different “O” code (Operational Code) is selected(i.e. 0x5) which is different from that for the sequence ordered setused by Ethernet or the signal ordered set used by Fibre channel. Theinformation to be transmitted in the FlexE overhead is encoded into thebytes D1, D2, and D3 of the overhead set block is shown in FIG. 9.

The information which needs to be included in the overhead includes:

-   -   The number of PHYs 22 in the FlexE group 12;    -   The identity (sequence) of this PHY 22 within the FlexE group        12;    -   A way to transmit the programming of the calendars from the        FlexE mux to the FlexE demux;    -   A way to indicate which calendar (“A” or “B”) is in use at this        time; and    -   A management channel. This may not be necessary in all        applications (for example, if a network management system has        direct access to the FlexE shim 16 at both ends of the        connection), but may be useful for applications such as using        FlexE for an n×100G umbilicus to a remote shelf of lower-rate        ports.

The amount of information to be conveyed from the FlexE mux to the FlexEdemux exceeds the 24 bits available. This is addressed by spreading therelevant overhead across a sequence of four FlexE overhead blocks, eachseparated by 20×1023 FlexE data blocks. The encoding of the informationin the sequence of four FlexE overhead blocks is illustrated in FIG. 10.The meaning, interpretation and processing of this overhead is explainedin clause 7 of Contribution Number: oif2015.127.01, the entirety ofwhich is incorporated herein by reference. The first of each sequence offour FlexE overhead blocks is encoded as an ordered set as shown in FIG.9. The next three FlexE overhead blocks are encoded as data 66b blocksat known locations based on the position of the ordered set block. Theblock with the higher order 2 bits of D1=00 serves as a marker to beused for alignment and re-interleaving of the sub-calendars from each ofthe PHYs 22 of the FlexE group 12 at the FlexE demux. The full sequencerepeats once per approximately 52.4 μs. Subject to the amount of bufferprovided in a given implementation, skew detection and compensationacross the PHYs 22 of the FlexE group 12 can be compensated up to a bitless than half of this amount.

FlexE Group Functions

The FlexE group 12 includes from 1 to n 100GBASE-R PHYs 22. Each100GBASE-R PHY 22 reuses nearly all of the functions as described for100GBASE-R in IEEE Std 802.3-2015. This includes a subset of thefunctions of the PCS as described in clause 82, and all of the functionsfrom sub-layers below the PCS as described in clauses 83, 86, 88, and91-95 as appropriate according to the PHY type.

The FlexE shim 16 provides to each FlexE group PHY 22 a set of 64b/66bencoded blocks that are encoded according to FIG. 82-4 in IEEE Std802.3-2015. Within the PCS, clause 82, each FlexE group PHY reuses (withreference to FIG. 82-2 in IEEE Std 802.3-2015) in the transmitdirection, the scrambler, block distribution, and alignment insertionprocesses. In the receive direction, each FlexE group PHY reuses thelane block sync, alignment lock, and lane deskew (including Bit ErrorRate (BER) monitor), lane reorder, alignment removal, and descrambling.

FlexE Clients

The format and standard bit rate of FlexE client 14 are described inclause 6.2 in IEEE Std 802.3-2015. However, FlexE also introduced theconcept of flexible non-standard rates such as n×25G (i.e. 175G, 225G,300G . . . ). FlexE clients 14 generally originate from one of thefollowing sources. A FlexE client 14 may be generated internally withina system, for example from a Network Processing Unit (NPU) within arouter or the like. The packet flow is generated at the determined FlexEclient 14 MAC rate and 64b/66b encoded according to IEEE Std 802.3-2015FIG. 82-4.

FlexE clients 14 at the rates of 10G, 40G, 100G, and in the future 25Gand 400G can be created from an Ethernet PHY 22 at the correspondingrate with some processing to convert to the FlexE client 14 format andrate.

A 10GBASE-R signal will be converted to a 10G FlexE client signal formatbefore presenting to a FlexE mux by using the idle insertion/deletionprocess as described in IEEE Std 802.3-2015 clause 82.2.3.6 (which willactually be doing idle deletion) to adapt the signal to the 10G FlexEclient rate and align start of packet to an 8-byte boundary, encodingaccording to the 66b block format of FIG. 82-4 in IEEE Std 802.3-2015from the received format which uses the blocks according to FIG. 49-7 inIEEE Std 802.3-2015. A 10G FlexE client signal coming from a FlexE demuxcan be converted to a 10GBASE-R signal format by using the idleinsertion/deletion process as described in IEEE Std 802.3-2015 clause49.2.4.7 (which will actually be doing idle insertion to compensate forthe space that had been occupied by FlexE group lane alignment markersand FlexE overhead), which also aligns to 4-byte boundaries, and encodesthe blocks according to FIG. 49-7 in IEEE Std 802.3-2015.

The 25GBASE-R specification is under development in the IEEE P802.3byproject, which is incorporated herein by reference. While thespecification has not been finalized, judging from currently adoptedbaselines, converting a 25GBASE-R signal to a 25G FlexE client signalformat is expected to involve correcting FEC errors (if FEC present),removing the FEC, removing the CWM (if present), trans-decoding to64b/66b, and using the idle insertion/deletion process as described inIEEE Std 802.3-2015 clause 82.2.3.6 (which will actually be doing idledeletion to make room for the FlexE overhead) to adapt the signal to the25G FlexE client rate and align start of packet to an 8-byte boundary,encoding according to the 66b block format of FIG. 82-4 in IEEE Std802.3-2015 from the received format which uses the blocks according toFIG. 49-7 of the same standard. The conversion of a 25G FlexE clientsignal coming from a FlexE demux to a 25GBASE-R signal is expected toinvolve using the idle insertion/deletion process as described in IEEEStd 802.3-2015 clause 49.2.4.7 (which will actually be doing idleinsertion to compensate for the space that had been occupied by FlexEoverhead—the FlexE group lane alignment markers take the same proportionof the space as the CWM), 256b/257b transcoding, insertion of the CWM,and calculation and insertion of FEC, if appropriate.

A 40GBASE-R signal can be converted to a FlexE client 14 by serializingand deskewing the PCS lanes, removing the PCS lane alignment markers,and using the idle insertion/deletion process as described in IEEE Std802.3-2015 clause 82.2.3.6 (which will actually be doing idle deletionto make room for the FlexE overhead) to adapt the signal to the 40GFlexE client rate. A 40G FlexE client 14 coming from a FlexE demux isconverted to a 40GBASE-R interface by using the idle insertion/deletionprocess as described in IEEE Std 802.3-2015 clause 82.2.3.6 (which inthis direction is generally doing idle insertion to restore the nominalbit-rate after removal of FlexE overhead), distributing the blocksround-robin to the four PCS lanes, and inserting PCS lane alignmentmarkers.

A 100GBASE-R signal without FEC can be converted to and from a FlexEclient 14 in the same manner as 40GBASE-R described above (except thatthe number of PCS lanes is 20 rather than 4). A 100GBASE-R signal withFEC, in converting to a FlexE client 14, also will correct any errorsper the FEC code, remove the FEC, and trans-decode from 256b/257b priorto the idle insertion/deletion process. To convert a 100G FlexE client14 coming from a FlexE demux to a 100GBASE-R signal with FEC involvesthe same processes as for 40GBASE-R, but in addition, transcoding thesignal to 256b/257b, inserting the FEC lane alignment markers, andadding the FEC.

400GBASE-R is under development in the P802.3bs task force. Thisstandard is not complete, but the process of converting to and from a400G FlexE client 14 is expected to be similar to that for converting a100GBASE-R client with FEC to and from a 100G FlexE client 14.

A FlexE client might come from another FlexE shim. In the case oftransport network equipment which implements the FlexE shim 16, FlexEclients 14 can be delivered from the FlexE shim 16 at the transportnetwork ingress to another FlexE shim at the transport network egress.The FlexE flow as a sequence of 64b/66b encoded blocks is expected to becarried over the transport network without packet loss. As no timinginformation is carried by this stream, idle insertion or padding ispossible in the mapping over the transport network. The FlexE shim 16 atthe network egress will only need to perform idle insertion/deletionaccording to IEEE Std 802.3-2015 clause 82.2.3.6, not due to anyexpected change in the nominal bit-rate, but simply to align the clockwith that of the other FlexE clients 14 for the same shim.

Interconnect Flexibility

Note that since the format of the FlexE client 14 is simply a logicallyserial stream of 66b blocks at a given rate, FlexE clients 14 do notneed to be produced or received in the same manner at both ends of theconnection. For example, a 10G, 25G, or 40G FlexE client 14 might begenerated as a system internal signal in the main chassis of a system,connected using an n×100G FlexE umbilicus to a satellite shelf, andconnected to physical 10GBASE-R, 25GBASE-R, and 40GBASE-R ports on thesatellite shelf. In the case where the FlexE mux is receiving a FlexEclient 14 from a physical Ethernet port and the FlexE demux isdelivering that FlexE client 14 to a physical Ethernet port, the twoports have to be the same rate, but they do not have to be the same PHYtype.

FlexE Overhead

The format of the FlexE overhead is indicated in FIG. 10. The FlexEoverhead is encoded as a 66b block and is inserted on each PHY 22 of theFlexE group 12. One overhead block is inserted after every 1023iterations of the length 20 calendar of FlexE data blocks, so thesequence is one block of overhead followed by 1023×20 blocks of datafollowed by one block of overhead.

A FlexE overhead lock is achieved at the receiver (FlexE demux) byrecognizing the FlexE “O” code used to send the first of every fourFlexE overhead blocks, so the sync header is 10, the control block typeis 0x4B (ordered set), and the “O” code is 0x5. Once FlexE overhead lockis achieved, the next expected FlexE overhead will be 1023×20+1 blockpositions later. While in FlexE overhead lock, bytes D1-D3 of theordered set block, plus the data blocks occurring at 66 B blockpositions 20461, 40922, and 122766 blocks beyond the ordered set blockwill be interpreted as FlexE overhead multiframe. FlexE overhead is notinterpreted if not in FlexE overhead lock. FlexE overhead lock will belost if the sync header, control block type, or O code do not match atthe expected position, such as for 5 occurrences.

Calendar

There are two calendars for each PHY 22 of the FlexE group 12: the “A”calendar (encoded as 0) and the “B” calendar (encoded as 1). The twocalendars are used to facilitate hitless reconfiguration. Normally,changes are only made to the calendar which is not currently in use.Exceptions would include initial link configuration or replacement of afailed circuit pack where it is necessary to download the calendarinformation into the replacement pack.

The calendar in use is signaled from the FlexE mux to the FlexE demux inbit position 2 of the first three of the four codewords in the FlexEmultiframe sequence. While most of the FlexE overhead can be reliablyprotected by the Cyclic Redundancy Check (CRC), the calendar in use mustbe interpreted even if the CRC is bad, since the FlexE demux must switchits calendar in use at precisely the same multiframe boundary as theFlexE mux. So that this can be done reliably, three copies of thecalendar in use are transmitted and interpreted by the receiver bymajority vote. Since the three copies are separated into different FlexEoverhead blocks across the multiframe (1,351,704 bits apart from eachother), the different copies will never be affected by the same bursterror. Since each PHY should have a BER of 10⁻¹² or better, theprobability of two instances of the calendar in use being wrong is nomore than 10⁻²⁴, which can safely be ignored.

When the calendar in use changes from a 0 to a 1, or from a 1 to a 0,the calendar used by both the FlexE mux and the FlexE demux will bechanged beginning with the first FlexE data block of the next FlexEoverhead ordered set block.

The set of PHYs 22 in the FlexE group 12 (not necessarily consecutive)are indicated in the “PHY Map Active” portion of the PHY Map field ofthe FlexE overhead, distributed over the same group of forty sets offour blocks of FlexE overhead as are used to transmit the contents ofthe two calendars as described herein. The identity of this PHY 22 inthe FlexE group (from 1 to n) is encoded in the D3 byte of the FlexEoverhead ordered set block. Note that this is persistent informationwhich does not change frequently. The receiver accepts a value for “ThisPHY” when seen identically for the second time in frames with a goodCRC, and updates to the PHY map bit map are accepted from frames withgood CRC. The “PHY Map Request” bits of the PHY map are used for aprotocol for the addition and removal of PHYs from the FlexE servergroup as described herein.

The contents of both the A and B calendar are transmitted continuouslyfrom the FlexE mux to the FlexE demux, with one calendar slot beingprogrammed in each multiframe of the FlexE overhead. The PC bitindicates which calendar (0=the “A” calendar and 1=the “B” calendar).All slots of the A calendar are programmed (20 muliframes, one for eachslot), followed by all slots of the B calendar (20 multiframes). The PCbit will be equal to 0 for the twenty multiframes for programming of the“A” calendar, and equal to 1 for the twenty multiframes for programmingof the “B” calendar. The receiver (FlexE demux) achieves calendar lockwhen the PC bit changes from 0 to 1 or from 1 to 0 in consecutivemultiframes with a good CRC. The receiver loses calendar lock if theexpected value of the PC bit is wrong in a multiframe with a good CRC.

The calendar is programmed in the same order as the calendar is in theframe from slots 0 to 19, starting with the first multiframe where PC=0(or is expected to be zero while in calendar lock) for the A calendarand the first multiframe where PC=1 (or is expected to be 1 while incalendar lock) for the B calendar.

The Client field indicates which of the FlexE clients is mapped into agiven calendar slot. The size of a given FlexE client can be calculatedbased on the number of calendar slots that client is assigned to. TheClient is a 16-bit field transmitted in the 2^(nd) and 3^(rd) octets inthe second “row” of the multiframe (the first data block after theordered set). The value 0x000 indicates a calendar slot which is unused(but available). The value 0xFFFF (all ones) indicates a calendar slotwhich is unavailable, for the case indicated in FIG. 5 where the fullFlexE group PHY rate cannot be carried over the transport network.

Calendar slot assignment changes are ignored while not in Calendar lock.Any change to which client is mapped into a calendar slot is ignored ina frame with a bad CRC. The full contents of both calendars aretransmitted from the FlexE mux to the FlexE demux approximately onceevery 2 ms. The fact that the calendars are transmitted continuouslyavoids an inconsistency between the calendars at the FlexE mux and theFlexE demux due to a lost message.

The normal process of reconfiguration (e.g., adding new FlexE clients 14to the FlexE group 12 or resizing a FlexE client 14) will involveprogramming the new configuration into the calendar which is not in use,then switching to the updated calendar, and finally updating theoriginal calendar. The FlexE mux should wait for at least 3 cycles oftransmission of the full calendar after the last update before switchingthe updated calendar to be the new calendar in use as described inclause 7.3.2 in IEEE Std 802.3-2015. This provides a vanishingly smallprobability that a calendar update has not been successfully receivedwithout requiring an Acknowledgement (ACK).

Management Channel

Certain applications may require the use of a management channel betweenthe FlexE mux and the FlexE demux. The management channel may not berequired for an application where a single management system orcontroller has access to the FlexE shims 16 on both ends of the FlexEgroup 12, but certain applications such as using FlexE for an n×100Gumbilicus to a remote shelf may use the management channel forcommunication with the controller in the remote shelf for configuration,alarming, software update, etc. When the management channel is not used,it is transmitted as zeros before scrambling

The format of the management channel is not specified and is applicationspecific. The management channel occupies 18 bytes of each FlexE. Thetotal capacity of the management channel is approximately 2.749 Mb/s.

Each PHY 22 of the FlexE group can carry its own management channel. Themanagement channels are not aggregated across the FlexE group to keepopen the possibility to pass back remote failure indications, forexample, losing one of the PHYs 22 of the FlexE group 12. Most likelyprotocols that would be carried over this type of channel would bemessage oriented and would have some sort of frame check sequence oneach packet, and therefore not require marking bytes of the managementchannel as bad if the CRC on the FlexE overhead is bad.

FlexE Group Number

For some applications, it may be desirable to be able to distinguish oneFlexE group 12 from another: for example, an 8-port device which iscapable of being provisioned as a single 8-port group or two 4-portgroups may create a situation where the PHY numbers could overlapbetween different groups. For such a situation, a 7-bit FlexE groupnumber is provided which allows for checking that the correct PHY 22 isbeing received from the correct group number. When this field is used,it must be provisioned to the same value in both directions. When anon-zero value is provisioned, the received group number will be checkedagainst the provisioned group number, and any mismatch will be alarmedto indicate the misconnection.

Reserved Bits

18 bits per FlexE overhead multiframe are reserved for possible futureextensions to this implementation. The reserved bits can be transmittedas zero before scrambling. An implementation could choose to treatreceipt of non-zero in the reserved bits in a multi-frame with a goodCRC as a minor alarm, as it could indicate interconnection with a futureversion of this implementation that is not known to be interoperable. Animplementation could also choose to ignore these bits on receipt andleave the responsibility to an implementation of a newer version of theimplementation to recognize receipt of zeros as an indication ofinterconnection with an older version, and presumably the newer versionknows whether it is interoperable with the older version.

CRC-16

Primarily to avoid corrupting the content of the calendars in thepresence of bit errors, the FlexE overhead is protected by a CRC. TheCRC is calculated over the following bits across the four rows of theFlexE overhead multiframe (in the order transmitted and received, notthe order described):

-   -   The D1, D2, and D3 bytes of the ordered set overhead block    -   All eight octets after the sync header of the 2^(nd) and 3^(rd)        overhead data blocks    -   The first six octets after the sync header of the 4^(th)        overhead data block.

The CRC is calculated using the polynomial x¹⁶+x¹²+x⁵+1. This value isinserted by the FlexE mux into the transmitted overhead. It iscalculated by the FlexE demux over the same set of bits and compared tothe received value. Various overhead described in the previous clausesis either accepted or ignored based on whether the CRC matches theexpected value.

FlexE Mux Data Flow

The FlexE Mux creates a logically serial stream of 66b blocks byinterleaving FlexE client signals, according to a master calendar oflength 20 n slots for a FlexE group including n 100GBASE-R PHYs. Eachslot corresponds to 5G of bandwidth. A FlexE client is assigned a numberof slots according to its bandwidth divided by 5G. The master calendaris distributed as described earlier in FIG. 6.

FIG. 11 presents an example of insertion of different bandwidth FlexEclients 14 into a logical master calendar. The slots assigned to aparticular FlexE client 14 do not all need to be on the same PHY 22 ofthe FlexE group 12, and new clients 14 can be added as long as there aresufficient slots available. Any slot in the master calendar which iseither “unassigned” or “unavailable” will be filled with Ethernet Errorcontrol blocks with the format given in FIG. 12.

This ensures that any error in calendar slot assignment cannot appear tothe FlexE demux as valid FlexE client data. Any incoming FlexE client 14which has failed (e.g., one created from a physical Ethernet signalwhere the signal has failed, or failure to achieve block lock) willcause the slots assigned to that FlexE client 14 to be filled withEthernet Local Fault ordered sets with the format as given in FIG. 13.

These rules allow for the creation of the complete data sequence on eachPHY 22 of the FlexE group 12. The FlexE overhead is inserted onto eachFlexE group PHY after every 1023 repetitions of the calendar sequence inthe same relative position to the calendar sequence on every PHY 22.This provides a marker which allows the data from the different PHYs 22of the FlexE group 12 to be re-interleaved in the original sequence sothat the FlexE clients 14 can be extracted. The 66b block stream is thenconverted into the format for the individual FlexE group PHY, whichincludes block distribution and alignment marker insertion, along with(if applicable) 256b/257b transcoding and FEC calculation and insertion.

FlexE Demux Data Flow

The FlexE Demux operates on a sequence of 66b blocks received from eachPHY 22 of the FlexE group 12. Recovering this sequence of blocksincludes (if applicable), FEC error correction and FEC remove andtrans-decoding to 64b/66b, PCS or FEC lane alignment, reinterleaving,and alignment marker removal. Once this has occurred, the PHYs 22 of theFlexE group 12 are re-interleaved so that FlexE clients 14 can berecovered as illustrated in FIG. 14. Note that the FlexE overheadmultiframe repeats on a cycle of approximately 52.4 μs, which allowsmeasuring skew differences between PHYs 22 of the FlexE group 12 ofapproximately ±25 μs.

Skew Tolerance Requirements

The amount of skew to be expected between the PHYs 22 of the FlexE group12 are application specific. Skew requirements are specified for twoclasses of applications—low skew and high skew applications. Low SkewApplications include intra-data-center applications, plus thosetransport network applications where the FlexE shim 16 is implemented inthe transport equipment, and the FlexE clients 14 rather than the PHYs22 of the FlexE group 12 are carried across the transport network. Theskew tolerance requirement for low skew applications is about 300 ns.Note that the intra-PCS-lane skew tolerance requirement for 100GBASE-Ris about 49 ns. A larger skew budget is established for FlexEapplications of similar reach to account for the fact that the PCS lanedeskew is not synchronized across the PHYs 22 of the FlexE group 12, andthere may be other variation, such as cable length, or evenheterogeneous 100GBASE-R PHY types which are not present in a single100GBASE-R interface.

High Skew Applications include transport network applications where thePHYs 22 of the FlexE group 12 rather than the FlexE clients 14 arecarried over the transport network (FlexE unaware transport). The skewtolerance requirement for high skew applications is about 10 μs. This isestablished to account for about 6 μs of dispersion-related skew if thePHYs are mapped over lambdas at opposite ends of the “C” band over largedistances (e.g., trans-pacific), with extra margin for things likesplit-band amplifiers and patch cords or the processing time to crunchand uncrunch the signal in the case where not all of the calendar slotscan be carried over the transport network connection.

If the intra-PHY skew exceeds the skew tolerance, the FlexE clients 14will not be demapped from the incoming PHYs 22, but will be sentcontinuous Ethernet Local Fault Ordered sets as illustrated in FIG. 13at the FlexE client rate. If one or more of the PHYs 22 of the FlexEgroup 12 has failed (e.g., loss of signal, failure to achieve block lockor alignment lock, high BER, or any other condition that results inPCS_Status=FALSE), the FlexE clients 14 will be sent continuous EthernetLocal Fault Ordered sets as illustrated in FIG. 13 at the FlexE clientrate. In addition, when one or more of the PHYs 22 of the FlexE group 12have failed continuous remote fault ordered sets will be sent back overthe FlexE group 12 in the calendar slots assigned to any FlexE clients14, and the RF bit will be set in the FlexE overhead of every PHY in theFlexE group toward the far-end shim.

Addition and Removal of PHYs from a FlexE Group

In general, only one PHY 22 should be added to or removed from a FlexEgroup 12 at a time. An added PHY 22 begins with all of its calendarslots (except those provisioned as unavailable) as unused. Prior toremoving a PHY 22, all clients must be removed from all of its calendarslots in both the “A” and the “B” calendar, i.e., all calendar slotswhich are not provisioned as “unavailable” will be “unused”. Additionand removal of PHYs does not change the (PHY,slot) identifier for anycalendar slot on any other PHY, and does not change the order ofcalendar slots assigned to any currently provisioned FlexE client 14.

Addition and removal of PHYs are controlled by the “PHY map active” and“PHY map request” fields of the FlexE overhead. Changes to either ofthese fields are only accepted from overhead multiframes with a goodCRC. These fields are sent identically on every current PHY of the FlexEgroup. The receiver may choose to receive these fields from multiplePHYs (so if a CRC is bad on one PHY, it receives it from another), ormay choose to receive the overhead from only one PHY and rely on thefact that the overhead is repeated in a continuous cycle to receiveeventually updates through repetition.

Whether the PHY addition procedure to add a new PHY to the group isinitiated from one or both ends is application dependent. The new PHYmight be added at both ends by the management or control system, or oneend might be the master and the other the slave, with the slave echoingback the PHY addition request initiated by the master. The initiationprocedure begins by changing the bit position in the “PHY map requestfield” from a zero to a one, bringing up the new PHY, and transmittingthe FlexE overhead on the PHY to be added.

The PHY is considered added once the following conditions are met:

-   -   The far end has also changed the bit position for the PHY to be        added from 0 to 1 in the “PHY map request” field.    -   PCS_status=TRUE for the PHY to be added (no Loss of Signal        (LOS), block lock and AM lock have been achieved on every PCS        lane, no hi_ber).    -   FlexE overhead lock and calendar lock are achieved on the added        PHY, the expected value is being received in the “This PHY”        field, and the FlexE overhead being received is within the skew        tolerance with the rest of the FlexE group.

Once these conditions are met, the bit position corresponding to theadded PHY is changed from a 0 to a 1 in the “PHY map active” field.FlexE clients 14 may not be assigned to the newly added PHY until boththe near end and the far end are indicating a one in the “PHY mapactive” field for the new PHY.

For PHY removal, all FlexE clients 14 must be removed from the A and theB calendars on a PHY before it can be removed. Any calendar slot notprovisioned as “unavailable” must be “unused” in both calendars on thePHY. Whether the initiation procedure occurs from one or both ends isimplementation dependent—both ends could be configured, or thecontroller at the far end could echo back the near end request in amaster/slave configuration. The procedure begins by changing the PHY MapRequest bit corresponding to the PHY to be deleted from a 1 to a 0.After the near end has changed the PHY Map Request bit to a 0, when thefar end also has changed its PHY Map Request bit to a zero, the near endchanges its PHY Map Active bit to a 0. The PHY is considered deletedonce a 0 is being received in the PHY Map Active bit from the far end.The FlexE overhead continues to be sent on the PHY being deleted untilthe far end is indicating 0 in its PHY Map Active bit corresponding tothe PHY.

FlexE Management

For FlexE group management, the control variables are the number of PHYs22 in the FlexE group. The status variables are PHY_align—True if allPHYs in the FlexE group are being received and are within the skewtolerance of the implementation. False if one or more of the incomingPHYs has failed or if the inter-PHY skew exceeds the skew tolerance ofthe implementation.

For FlexE group management, per PHY, note that all 100GBASE-R controland status variables in or blow the PCS are applicable to each PHY ofthe FlexE group 12.

Control Variables:

-   -   PHY number (within the FlexE Group)    -   Number of available Calendar Slots (20 except when entire PHY        can't be carried across the transport network)    -   Mux_Calendar_A[0 . . . 19] and Mux_Calendar_B[0 . . . 19] The        FlexE client that is mapped into each of the respective calendar        slots in the transmit direction. The initial default for all        calendar slots is UNUSED, except in the case where the number of        available calendar slots is less than 20, in which case the        first AVAILABLE number of slots default to UNUSED and the rest        default to UNAVAILABLE.    -   Tx_Calander_in_use—Indicates whether the A or the B calendar is        currently active.

Status Variables

-   -   Demux_Calendar_A[0 . . . 19] and Demux_Calendar_B[0 . . . 19]        The FlexE client that the far end has mapped into each of the        calendar slots in the received direction. The initial default        for all calendar slots (prior to information received from the        far end) is UNUSED, except in the case where the number of        available calendar slots is less than 20, in which case the        first AVAILABLE number of slots default to UNUSED and the rest        default to UNAVAILABLE.    -   Rx_Calendar_in_use—Indicates the calendar being used at the        current time by the far end.

Transport Network Mappings for Flex Ethernet Signals

Three different methods of mapping of FlexE signals over transportnetworks are possible. The case of FlexE unaware transport involves thetransport network mapping each of the Ethernet PHYs is independentlyover the transport network using a PCS codeword transparent mapping.Since the FlexE mux and FlexE demux are separated by transport networkdistances, this requires a “high skew” implementation of the FlexE shim16.

Note that certain existing OTN mappers/demappers are not fully PCScodeword transparent with respect to LF and RF ordered sets, and maymistake an LF or RF sent for an individual FlexE client as an LF or RFfor the entire link and bring the link down. This is not an issue in thecase of FlexE unaware transport of simple bonding to carry a larger rateflow, as a failure of the single high-rate FlexE client is equivalent toa failure of the group. But it may be an issue if FlexE unawaretransport is used to carry a group of multiple lower-rate FlexE clientservices using less than fully PCS codeword transparent mappings.

The next case is where the FlexE shim 16 is implemented in the transportnetwork equipment, and rather than carrying the PHYs 22 of the FlexEgroup 12 over the transport network, the FlexE clients 14 are carriedover the transport network. Since the 64b/66b encoded FlexE client rateis:

${{FlexE}\mspace{14mu} {Client}\mspace{14mu} {MAC}\mspace{14mu} {rate} \times \frac{66}{64} \times \frac{16383}{16384} \times \frac{20460}{20461}} \pm {100\mspace{14mu} {ppm}}$

Note that while this is the stream of blocks presented to the transportnetwork from the FlexE shim 16 to be carried, since this stream does nothave any timing information, the transport network is not required tosignal at this rate: idle insertion/deletion or padding may be used inthe mapping specified by ITU-T if it provides a more convenient rate forthe transport.

When a FlexE client 14 is mapped in this manner, it may be connected atthe OTN egress to another FlexE shim 16 where it will be clock alignedwith other FlexE clients 14 carried over the same FlexE group 12. It mayalso be connected to an Ethernet PHY 22 with the same nominal MAC rateas the FlexE client 14 using the appropriate conversion as describedherein.

Note that in the case where it is necessary to connect a FlexE client 14to an Ethernet PHY 22 across an OTN where the Ethernet PHY 22 uses alegacy mapper/demapper, it may be necessary to perform the conversion ofthe FlexE client 14 to the Ethernet PHY format according to clause 7.2.2immediately after the FlexE shim 16 and to map the FlexE client 14 overOTN as if it were an Ethernet PHY 22 of the corresponding rate. Thissecond case can use a “low skew” implementation of the FlexE shim 16 asdescribed herein.

The third case is where the FlexE shim 16 is not terminated in thetransport network equipment, but the transport network equipment“crunches” the PHY 22 of the FlexE group 12 by discarding unavailablecalendar slots at the transport network ingress and re-inserting them atthe transport network egress. The mapping of this requires serializingand deskewing the PCS lanes of the PHY, then discarding all“UNAVAILABLE” calendar slots from the structure to reduce the bit-rate.For example, if only 15 of 20 calendar slots are available, there areeffectively 1023 repetitions of a length 15 calendar after discardingthe unavailable slots. At the transport network egress, the unavailableslots are restored to the FlexE PHY stream of 66b blocks (insertingerror control blocks as illustrated in FIG. 12). The net bit-rate ofthis reduced-rate flow when there are “n” available calendar slots is:

${103.125\mspace{14mu} {Gb}\text{/}s \times \frac{16383}{16384} \times \frac{1 + {1023n}}{20461}} \pm {100\mspace{14mu} {ppm}}$

FlexO and FlexE

Referring to FIG. 15, in an embodiment, a diagram illustrates acomparison between the layer stack and atomic functions of FlexO andFlexE. Of note, FlexO and FlexE are converging and look alike in thelayer stack and atomic functions. In various embodiments, the systemsand methods described herein contemplate FlexE as becoming the nextB100G multiplexed structure. For example, map Low Order (LO) Opticalchannel Data Unit-j (ODUj), High Order (HO) Optical channel Data Unit-k(ODUk) and ETY right into a FlexE shim 16. The FlexE group 12 can be asection layer, and the FlexE client 14 (the 64b/66b stream, which couldbe OTN) can become a switchable Operations, Administration, andMaintenance (OAM) service.

FlexE Service OAM

Referring to FIG. 16, in an embodiment, a network diagram illustrates acomparison of FlexO service OAM with the FlexE Client service OAMsystems and methods described herein. Specifically, the same equipmentis shown in FIG. 16 for FlexO and FlexE, including a transport network100 with network elements 102, 104 coupled to switches/routers 106, 108.In an embodiment, the FlexE Client OAM systems and methods enable aFlexE client to be transported as a service (or private line service) ina provider network. The switches/routers 106, 108 can be referred to assubscriber equipment where a service originates or terminates, i.e., amaintenance entity. In FlexO, the service can be an ODUflex, and, inFlexE, the service can be a FlexE client. The network elements 102, 104are communicatively coupled to one another. In FlexO, theinterconnection between the network elements 102, 104 and theswitches/routers 106, 108, respectively, can be m×OTUCn, and, in FlexE,the interconnection can be m×FlexE groups.

The transport network 100 can include a plurality of network elements inaddition to the network elements 102, 104. The various network elementsin the transport network 100 are configured to performtransport/switching. In FlexO, the transport/switching can be atODUk/flex or some other variant of OTN. In FlexE, thetransport/switching can be ODUk/flex and/or FlexE-based switching. Anexample of FlexE-based switching is described in U.S. Patent ApplicationNo. 62/186,489, filed on Jun. 30, 2015, and entitled “FLEXIBLE ETHERNETSWITCHING SYSTEMS AND METHODS,” the contents of which are incorporatedherein by reference. In FlexO, the network elements 102, 104 provideUser-Network Interface (UNI)/Inter-Domain Interface (IrDI)/ExternalNetwork-Network (E-NNI) interfaces. Conversely, the FlexE Client OAMsystems and methods enable FlexE UNI/IrDI interfaces at the networkelements 102, 104.

In an embodiment, the network 100 can support a FlexE client, and thenetwork 100 can include a plurality of nodes interconnected to oneanother, wherein the FlexE client is configured between two nodes;wherein each of the two nodes include circuitry configured to receivethe FlexE client, and circuitry configured to update or monitor one ormore Operations, Administration, and Maintenance (OAM) fields in FlexEoverhead, wherein the OAM fields cover a single client path for theFlexE client. One of the two nodes can be a service terminatingsubscriber equipment and another of the two nodes can be a networkelement in a transport network, wherein the FlexE client can include aUser-Network Interface (UNI). Each of the two nodes can be a networkelement in a transport network, and each of the two nodes can beconfigured to operate as an OAM monitoring point for the FlexE client inthe transport network. The one or more OAM fields can include any of aclient service monitoring field, a connectivity/trace, link fault,remote fault, maintenance states, test signals, link neighbor discovery,and backwards fault and error indication.

FlexE Client Performance Monitoring

Referring to FIG. 17, in an embodiment, a network diagram illustrates anetwork with various FlexE Client OAM monitoring points. In FIG. 17, thetransport network 100 is illustrated with additional network elements110, 112, and each of the network elements 102, 104, 110, 112 can beconfigured as a monitoring point for FlexE service OAM. In Ethernet,current transport equipment typically relies on MAC CRC (layer 2) forfault isolation and performance monitoring. In FlexE, this becomescomplicated as FlexE introduces a variable amount of clients on asingle/multiple ports (groups) and a variable rate for the clients(Flexible MAC). The FlexE Client OAM systems and methods can include asimpler FlexE client service monitoring field (e.g., Bit InterleavedParity 8 (BIP8) or the like) that covers only the 64b/66b codes for asingle client path, i.e., a single error check per client, instead ofCRC at the packet level. This can be viewed analogously to the PM BIP8that covers an ODUflex service path. Such an approach with a FlexEclient service monitoring field simplifies monitoring and faultisolation. Nodes in a transport network could be 3 R regens (i.e.Regenerator Section (RS)-OTN) or switches (i.e., Multiplex Section(MS)-OTN or FlexE switches).

In an embodiment, a node configured to monitor a Flexible Ethernet(FlexE) client service in a network includes circuitry configured toreceive a FlexE client; and circuitry configured to monitor or updateone or more Operations, Administration, and Maintenance (OAM) fields inFlexE overhead, wherein the OAM fields cover a single client path forthe FlexE client. The one or more OAM fields can include a clientservice monitoring field which does not rely on packet Cyclic RedundancyCheck (CRC). The one or more OAM fields can include a client servicemonitoring field including a bitstream Bit Interleaved Parity (BIP)code. The one or more OAM fields can include any of aconnectivity/trace, link fault, remote fault, maintenance states, testsignals, link neighbor discovery, and backwards fault and errorindication. The one or more OAM fields can be disassociated from LocalFault and Remote Fault information for an associated FlexE PHY/group.The one or more OAM fields can be located in the FlexE overhead in amultiframe scheme. The one or more OAM fields can be implemented in aReconciliation Sublayer with an associated Operational code, differentfrom the one used for FlexE OH (or current IEEE/FC codes). The node canbe one of a Regenerator Section (RS) Optical Transport Network (OTN)switch, a Multiplex Section (MS) OTN switch, and a FlexE switch.

FlexE OAM Fields

The following table includes proposed FlexE OAM fields:

Bit Error Check (e.g., BIP8) Connectivity/Trace (e.g., Trail TraceIdentifier (TTI)) Link Neighbor Discovery (i.e. LLDP) Link Fault RemoteFault Maintenance States (e.g., locked, loop-back, test) Test signals(e.g., Pseudorandom binary sequence (PRBS)) for turn-up Backwards faultand error indication

Referring back to FIG. 2, the FlexE group Remote Fault (RF)/Local Fault(LF) is handled at the Reconciliation Sub layer (RS), above the FlexEshim. The FlexE OAM cannot reuse the group RF/LF for clients as thiswould flag group/PHY faults in the transport equipment. Instead, theFlexE OAM could define a new “0 code” (e.g., 0x9) for client RF/LF todisassociate from the group/PHY. Again, drawing analogies to the OTNimplementations, this is analogous to LO ODUk-AIS Path Monitoring (PM)relative to LO/HO Section Monitoring (SM).

Referring to FIG. 18, in an embodiment, a diagram illustrates an FlexEClient OAM implementation with FlexE overhead. Specifically, FIG. 18illustrates a first option for OAM where the fields are included in theFlexE Time Division Multiplexing (TDM) frame overhead. The FlexE TDMframe overhead has a portion for all calendar slots and paths. Forexample, the FlexE Client OAM could occupy some or all of the reservedportions of the FlexE TDM frame overhead. However, it might be tightfitting OAM overhead for up to 10 clients (per group) in these portions.As such, an option could be to multiframe the OAM overhead, analogous towhat OTN does with the HO OPUk path OH. The FlexE OAM could be switchedalong with the calendar slots to provide a path. Another option couldinclude reducing the management channel to provide more room for theFlexE Client OAM or reusing the management channel capacity with amessaged based OAM approach on a general communications channel.

Referring to FIG. 19, in an embodiment, a block diagram illustrates theFlexE overhead from FIG. 9 with a new “O” code for implementing FlexEOAM at the RS layer. This embodiment includes injecting a new “O” code(e.g., 0x9), 1/16 k occurrence and steal bandwidth of the client (e.g.,59 parts per million (ppm))—worst case standard Ethernet at 100G has 400ppm of idles in there. This can be used for a new client LF/RF and todifferentiate from group LF/RF (0x0), and this can include specific OAMper client stream, which is then mapped to calendar slots along with theclient stream. These OAM fields are tied to FlexE clients and notcalendar slots. This second option is analogous to LO ODUk PM overhead.

Referring to FIG. 20, in an embodiment, the logical diagram of FIG. 2illustrates a location of the first and second options in the IEEE 802.3stack. The first option is located in the FlexE shim in the frameheader, stealing OH bandwidth of the FlexE OH (O code 0x5). The secondoption is implemented in the RS layer creating the concept of a path,like an ODUflex, stealing bandwidth from the client with new O code(i.e. 0x9). This can be snooped without requiring knowledge of packetsand boundaries of FlexE client for performance monitoring and can beswitched along with client 64b/66b bitstream as a path layer.

FlexE Monitoring Process

Referring to FIG. 21, in an embodiment, a flowchart illustrates aprocess 200, implemented in a node, for monitoring a Flexible Ethernet(FlexE) client service in a network. For example, the node can includethe network elements 102, 104 or the switches/routers 106, 108. Theprocess 200 includes receiving a FlexE client (step 202); and at leastone of monitoring and updating one or more Operations, Administration,and Maintenance (OAM) fields in FlexE overhead, wherein the OAM fieldscover a single client path for the FlexE client (step 204). The one ormore OAM fields can include a client service monitoring field which doesnot rely on Cyclic Redundancy Check (CRC). The one or more OAM fieldscan include a client service monitoring field including a BitInterleaved Parity (BIP) code. The one or more OAM fields can includeany of a connectivity/trace, link fault, remote fault, maintenancestates, test signals, and backwards fault and error indication. The oneor more OAM fields can be disassociated from Local Fault and RemoteFault information for an associated FlexE group/PHY. The one or more OAMfields can be located in the FlexE overhead, possibly using a multiframescheme. The one or more OAM fields can be implemented in aReconciliation Sublayer with a new associated Operational code. The nodecan be one of a Regenerator Section (RS) Optical Transport Network (OTN)switch, a Multiplex Section (MS) OTN switch, and a FlexE switch.

It will be appreciated that some embodiments described herein mayinclude one or more generic or specialized processors (“one or moreprocessors”) such as microprocessors, digital signal processors,customized processors, and field programmable gate arrays (FPGAs) andunique stored program instructions (including both software andfirmware) that control the one or more processors to implement, inconjunction with certain non-processor circuits, some, most, or all ofthe functions of the methods and/or systems described herein.Alternatively, some or all functions may be implemented by a statemachine that has no stored program instructions, or in one or moreapplication specific integrated circuits (ASICs), in which each functionor some combinations of certain of the functions are implemented ascustom logic. Of course, a combination of the aforementioned approachesmay be used. Moreover, some embodiments may be implemented as anon-transitory computer-readable storage medium having computer readablecode stored thereon for programming a computer, group, appliance,device, etc. each of which may include a processor to perform methods asdescribed and claimed herein. Examples of such computer-readable storagemediums include, but are not limited to, a hard disk, an optical storagedevice, a magnetic storage device, a ROM (Read Only Memory), a PROM(Programmable Read Only Memory), an EPROM (Erasable Programmable ReadOnly Memory), an EEPROM (Electrically Erasable Programmable Read OnlyMemory), Flash memory, and the like. When stored in the non-transitorycomputer readable medium, software can include instructions executableby a processor that, in response to such execution, cause a processor orany other circuitry to perform a set of operations, steps, methods,processes, algorithms, etc.

Although the present disclosure has been illustrated and describedherein with reference to preferred embodiments and specific examplesthereof, it will be readily apparent to those of ordinary skill in theart that other embodiments and examples may perform similar functionsand/or achieve like results. All such equivalent embodiments andexamples are within the spirit and scope of the present disclosure, arecontemplated thereby, and are intended to be covered by the followingclaims.

What is claimed is:
 1. A node configured to support a Flexible Ethernet(FlexE) client service in a network, the node comprising: circuitryconfigured to interface a FlexE client; and circuitry configured tomonitor and update one or more Operations, Administration, andMaintenance (OAM) fields associated with the FlexE client, wherein theone or more OAM fields comprise a monitoring field that covers 64b/66bcodes for a path of the FlexE client.
 2. The node of claim 1, whereinthe monitoring field provides an error check for the FlexE clientinstead of a Cyclic Redundancy Check (CRC) at a packet level.
 3. Thenode of claim 1, wherein the monitoring field utilizes a bitstream BitInterleaved Parity (BIP) code.
 4. The node of claim 1, wherein the oneor more OAM fields comprise a plurality of a connectivity/trace, linkfault, remote fault, maintenance states, test signals, link neighbordiscovery, and backwards fault and error indication.
 5. The node ofclaim 1, wherein the one or more OAM fields have a different “O” codefrom standard Local Fault and Remote Fault information for an associatedFlexE group/PHY.
 6. The node of claim 1, wherein the one or more OAMfields use a multiframe scheme with the one or more OAM fields switchedalong with calendar slots over the path.
 7. The node of claim 1, whereinthe one or more OAM fields are located in the path in a ReconciliationSublayer.
 8. The node of claim 1, wherein the one or more OAM fields arelocated in a frame header in a FlexE shim.
 9. The node of claim 1,wherein the node is one of a Regenerator Section (RS) Optical TransportNetwork (OTN) switch, a Multiplex Section (MS) OTN switch, and a FlexEswitch.
 10. A method, implemented in a node, for supporting a FlexibleEthernet (FlexE) client service in a network, the method comprising:interfacing a FlexE client; and updating and monitoring one or moreOperations, Administration, and Maintenance (OAM) fields associated withthe FlexE client, wherein the one or more OAM fields comprise amonitoring field that covers 64b/66b codes for a path of the FlexEclient.
 11. The method of claim 10, wherein the monitoring fieldprovides an error check for the FlexE client instead of a CyclicRedundancy Check (CRC) at a packet level.
 12. The method of claim 10,wherein the monitoring field utilizes a bitstream Bit Interleaved Parity(BIP) code.
 13. The method of claim 10, wherein the one or more OAMfields comprise a plurality of a connectivity/trace, link fault, remotefault, maintenance states, test signals, link neighbor discovery, andbackwards fault and error indication.
 14. The method of claim 10,wherein the one or more OAM fields have a different “O” code fromstandard Local Fault and Remote Fault information for an associatedFlexE group/PHY.
 15. The method of claim 10, wherein the one or more OAMfields use a multiframe scheme with the one or more OAM fields switchedalong with calendar slots over the path.
 16. The method of claim 10,wherein the one or more OAM fields are located in the path in aReconciliation Sublayer.
 17. The method of claim 10, wherein the one ormore OAM fields are located in a frame header in a FlexE shim.
 18. Themethod of claim 10, wherein the node is one of a Regenerator Section(RS) Optical Transport Network (OTN) switch, a Multiplex Section (MS)OTN switch, and a FlexE switch.
 19. A network supporting a FlexibleEthernet (FlexE) client, the network comprising: a plurality of nodesinterconnected to one another, wherein the FlexE client is configuredbetween two nodes and traverses one or more of the nodes; wherein eachof the two nodes comprise circuitry configured to interface a FlexEclient; and circuitry configured to monitor and update one or moreOperations, Administration, and Maintenance (OAM) fields associated withthe FlexE client, wherein the one or more OAM fields comprise amonitoring field that covers 64b/66b codes for a path of the FlexEclient.
 20. The network of claim 19, wherein the monitoring fieldprovides an error check for the FlexE client instead of a CyclicRedundancy Check (CRC) at a packet level.